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ABSTRACT 

This  presentation  will  provide  program  information,  goals  and  objectives  of  the  Technology  for  Human-Robot 
Interactions  in  Soldier-Robot  Teaming  (HRI)  Army  Technology  Objective  (ATO).  The  intent  of  this  program  is  to 
develop  and  demonstrate  an  intelligent  scaleable  interface  for  mounted  and  dismounted  control  of  ground  and  air 
unmanned  systems.  Currently  in  the  Army  there  are  unique  interfaces  developed  by  engineers  for  each  unmanned 
system  fielded.  This  saddles  the  soldier  with  a  training  burden  to  learn  specific  interface  operations  prior  to  controlling 
the  robot.  By  providing  a  consistent  look  and  feel  across  various  sized  controlling  devices,  the  training  burden  is 
reduced  as  well  as  the  soldier’s  cognitive  workload.  Additionally,  task  analysis  will  be  performed  to  identify  workload 
barriers  and  bottlenecks,  and  intelligent  agents  will  be  developed  and  applied  to  reduce  and/or  automate  the  higher 
workload  tasks.  Lastly,  this  program  will  develop  adaptive  automation  techniques  to  intelligently  shed  or  introduce 
tasks  at  the  appropriate  time  to  the  soldier  to  maintain  optimal  situational  awareness  and  maximize  the  performance  of 
the  soldier-robot  team. 
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1.  INTRODUCTION 

The  Technologies  for  Human-Robot  Interactions  (HRI)  in  Soldier-Robot  Teaming  Army  Technology  Objective  (ATO) 
(Fig  1)  was  established  in  2004  and  is  a  joint  program  between  the  Tank- Automotive  Research,  Development  and 
Engineering  Center  (TARDEC)  and  the  Army  Research  Laboratory’s  (ARL)  Human  Research  and  Engineering 
Directorate  (HRED),  both  agencies  under  the  Army  Materiel  Command’s  (AMC)  Research,  Development  and 
Engineering  Command  (RDECOM).  The  main  objective  for  the  program  is  to  provide  intelligent,  scalable  mounted  and 
dismounted  control  of  ground  and  air  unmanned  systems.  There  are  four  pacing  technologies  for  this  program; 
extensive  task  decomposition  and  analysis  to  establish  the  workload  of  the  soldier,  intelligent  agent  development  for 
offloading  or  automating  soldier  tasks  for  optimum  workload,  scalable  interfaces  for  presenting  all  relevant  information 
to  the  user  regardless  of  his  equipment  configuration  and  mission,  and  recursive  modeling  to  refine  and  validate  models 
developed  throughout  the  program.  The  ultimate  goal  is  to  reduce  the  controlling  workload  and  training  requirements 
of  unmanned  systems  to  permit  the  soldier  to  focus  solely  on  his  primary  mission.  Secondarily,  how  can  a  mounted  or 
dismounted  soldier  properly  interact  with  a  robotic  system  in  a  seamless  fashion  so  that  it  functions  as  another  member 

of  his  team? 

The  intent  of  this  paper  is  to  provide  background 
information,  a  brief  history  outlining  the 
establishment  of  the  HRI  ATO,  and  provide  an 
overview  of  technologies,  methodologies, 
approaches  and  support  applications  currently  under 
investigation  for  use  in  this  program. 

Section  two  of  this  paper  provides  a  program  history 
detailing  the  current  state  of  technologies  in  this 
area,  the  hurdles  identified  by  the  Army,  and  the 
methodology  to  be  employed  to  overcome  these 
barriers.  This  includes  near  term  and  long  term 
objectives  and  potential  transition  opportunities  for 

Figure  1:  Technologies  for  Human-Rohot  Interactions  (HRI)  in  Soldier- 
Robot  Teaming  Army  Technology  Objective  (ATO) 


Report  Documentation  Page 


Form  Approved 
0MB  No.  0704-0188 


Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 


1.  REPORT  DATE 

28  MAR  2005 


2.  REPORT  TYPE 

Journal  Article 


3.  DATES  COVERED 

29-10-2004  to  24-02-2005 


4.  TITLE  AND  SUBTITLE 

Human-Robot  Teaming 


6.  AUTHOR(S) 

Terrance  Tierney;  Brian  Samples;  William  Protzman 


7.  PEREORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

DCS  Corporation, 1330  Braddock  Place, Alexandria, VA, 22314 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

U.S.  Army  TARDEC,  6501  East  Eleven  Mile  Rd,  Warren,  Mi,  48397-5000 


5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

8.  PEREORMING  ORGANIZATION 
REPORT  NUMBER 

;  #14747 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

TARDEC 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

#14747 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

This  presentation  will  provide  program  information,  goals  and  objectives  of  the  Technology  for 
Human-Robot  Interactions  in  Soldier-Robot  Teaming  (HRI)  Army  Technology  Objective  (ATO).  The 
intent  of  this  program  is  to  develop  and  demonstrate  an  intelligent  scaleable  interface  for  mounted  and 
dismounted  control  of  ground  and  air  unmanned  systems.  Currently  in  the  Army  there  are  unique 
interfaces  developed  by  engineers  for  each  unmanned  system  fielded.  This  saddles  the  soldier  with  a 
training  burden  to  learn  specific  interface  operations  prior  to  controlling  the  robot.  By  providing  a 
consistent  look  and  feel  across  various  sized  controlling  devices,  the  training  burden  is  reduced  as  well  as 
the  soldier?s  cognitive  workload.  Additionally,  task  analysis  will  be  performed  to  identify  workload 
barriers  and  bottlenecks,  and  intelligent  agents  will  be  developed  and  applied  to  reduce  and/or  automate 
the  higher  workload  tasks.  Lastly,  this  program  will  develop  adaptive  automation  techniques  to 
intelligently  shed  or  introduce  tasks  at  the  appropriate  time  to  the  soldier  to  maintain  optimal  situational 
awareness  and  maximize  the  performance  of  the  soldier-robot  team. 

15.  SUBJECT  TERMS 

Interactions,  teaming,  soldier-machine,  robot,  scalable  interface,  intelligent  agents,  adaptive  automation 


16.  SECURITY  CLASSIEICATION  OE: 


a.  REPORT 

unclassified 


b.  ABSTRACT 

unclassified 


c.  THIS  PAGE 

unclassified 


17.  LIMITATION  OE 

18.  NUMBER 

ABSTRACT 

OE  PAGES 

Public  Release 

12 

19a.  NAME  OE 
RESPONSIBLE  PERSON 


standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


the  Future  Combat  Systems  (FCS)  program  and  the  Army  weapon  system  community  in  general. 

Section  three  of  this  report  provides  the  systems  engineering  approach  that  the  HRI  program  will  utilize  to  solve  human- 
robot  teaming  issues,  scalability  and  adaptive  automation  to  reduce  soldier  workload.  This  methodology  identifies 
soldier  requirements  and  delineates  tasks,  establishes  relevant  behaviors  of  the  system,  and  feeds  an  end  to  end 
modeling  system  to  test  and  validate  component  and  system  models. 

Section  four  provides  an  assessment  of  current  programs  and  technologies  under  investigation  for  incorporation  or 
leveraging  within  the  HRI  ATO  program.  These  programs  are  primarily  comprised  of  Army  ATO  and  Applied 
Technology  Demonstration  (ATD)  programs;  however  relevant  Army  acquisition  programs  are  also  included.  For  each 
program,  a  program  overview  along  with  relevant  technologies  and  potential  application  of  those  technologies  within 
the  HRI  ATO  program  is  summarized. 

Section  five  of  this  report  provides  a  technology  assessment  that  presents  an  overview  of  applicable  commercial  and 
science  and  technology  areas  of  interest  to  the  HRI  ATO  program.  The  technologies  presented  within  this  section  are 
anticipated  to  have  a  high  degree  of  applicability  to  the  HRI  ATO  and  will  be  addressed  and/or  rationalized  in  some 
fashion  within  the  resultant  ATO  products. 

Section  six  of  this  report  identifies  specific  conclusions  as  well  as  an  approach  for  continued  coordination  and 
interchange  of  information  and  products  between  the  HRI  ATO  and  the  FCS  programs  in  accordance  with  TTA  CT-I8. 

2.  PROGRAM  HISTORY 

In  2002,  the  Army  Science  Board  (Fig  2)  conducted  an  Ad  Hoc  Study^  on  human  robot  interfaces.  The  study  discussed 
the  emergence  of  unmanned  systems  in  the  military,  and  identified  some  of  the  current  shortfalls. 

In  particular,  the  study  observed  that,  “the  force  will  require  and  use  unmanned  systems  in  ways  that  involve  close 
interaction.  Further,  because  the  technology  required  for  full  autonomy  will  not  be  available  in  the  abbreviated  time 
frame  for  the  initiation  of  Army  transformation,  efforts  should  focus  on  controlled,  semiautonomous  operations.  These 
two  observations  lead  naturally  to  the  question  of  how  humans  and  unmanned  systems  interact.  In  this  study,  we 
focused  primarily  on  the  issues  surrounding  the  interactions  of  humans  with  unmanned  ground  systems.  The  ground 
environment  presents  significant  challenges  to  autonomous  systems,  in  large  part  due  to  the  navigation  requirements 
created  by  the  wide  variability  of  the  terrain,  and  the  close  proximity  between  the  autonomous  entities  and  humans  in 
the  environment.  These  challenges  to  automated  systems  operation  make  the  task  of  effective  human-robot  interactions 
particularly  important  to  mission  success.  Further,  airborne  vehicles  alone  cannot  efficiently  perform  a  large  number  of 
tasks  that  are  important  to  the  Army  necessitating  the  use  of  unmanned  ground  vehicles.” 

During  the  information  gathering  phase,  the  study  found  reasons  for  optimism 
as  well  as  areas  of  significant  concern.  Current  Army  programs  had  shown 
significant  strides  in  the  area  of  advanced  perception  relating  to  autonomous 
navigation,  but  that,  “no  existing  programs  systematically  approach  the 
challenges  of  interactions  between  humans  and  complex  unmanned  systems. 
Existing  literature  contains  numerous  examples  that  show  that  the  lack  of  rigor 
in  the  design  of  interactions  and  interfaces  between  humans  and  complex 
systems  can  lead  to  catastrophic  results  (e.g..  Three  Mile  Island,  the  USS 
Vincennes  shoot-down  of  the  Iran  Air  Airbus).  If  the  human-robot  interaction 
issue  is  not  systematically  addressed,  we  are  concerned  that  similar 
catastrophic  problems  could  arise  in  the  application  of  robotic  platforms  in  the 
Army.  This,  in  turn,  would  result  in  severe  setbacks  to  the  induction  of 
robotics  into  the  force.” 

The  study  concluded  with  three  basic  findings.  “First,  we  recommend  that  the  requirements  community,  led  by 
TRADOC  and  the  schools,  establish  an  operational  architecture  for  autonomous  robots,  and  validate  the  architecture 


through  an  aggressive  program  of  hands-on  usage  and  experimentation  with  available  robots  in  the  field;  (e.g.,  by  the 
Army  National  Guard,  by  the  Opposing  Force  (OPFOR)  at  the  National  Training  Center).  Second,  we  recommend  the 
creation  of  a  new  systems-oriented  program  for  the  analysis,  understanding,  development,  and  improvement  of  human- 
robot  interactions.  We  recommend  that  ARL,  in  cooperation  with  DARPA  and  other  technology  and  system  developers 
be  the  steward  for  such  a  program.  This  should  facilitate  technology  insights  and  lessons  learned  from  the  field  use  of 
robots  and  the  real  time  feedback  establish  the  baseline  for  future  developments;  a  process  that  should  promote  spiral 
development.  Finally,  we  recommend  that  the  Army  insist  that  FCS  Block  1  program  have,  at  a  minimum,  follower 
robots  with  a  significant  level  of  autonomy  and  surveillance  and  reconnaissance  robots  that  can  operate  in  limited 
environments — capabilities  that  can  developed  by  maturing  the  technology  that  exists  today.” 

Based  on  Army  Science  Board  recommendations  and  approval  from  ASAALT  and  TRADOC,  TARDEC  took  the  lead 
role  to  develop  a  systems  engineering  approach  in  solving  soldier-robot  interaction  issues,  teaming  with  ARL-HRED 
for  support  in  human  factors  engineering  concerns. 

3.  PROGRAM  METHODOLOGY 


TARDEC  and  ARL-HRED  are  teaming  to  solve  human-robot  interactions  for  mounted  and  dismounted  control  of 
ground  and  air  unmanned  systems.  The  approach  to  solving  this  problem  utilizes  an  end-to-end  systems  engineering 
approach  that  starts  with  the  identification  of  soldier  requirements  to  conduct  missions  when  teaming  with  unmanned 
systems.  These  requirements  are  defined  in  terms  of  the  tasks  to  be  performed,  in  turn  leading  to  the  identification  of 

system  behaviors.  System  behaviors  are 
captured  in  an  HRI  knowledgebase  in 
accordance  with  an  intelligent  system 
ontology,  along  with  the  technologies  and 
capabilities  associated  with  the  domain  of 
intelligent  systems  in  which  these 
behaviors  will  be  deployed.  To  the  extent 
possible,  the  behaviors  defined  within  the 
knowledgebase  will  be  specified 
independent  of  specific  technologies  or 
their  allocation  (i.e.,  hardware, 
architecture,  or  human  operators).  Once 
archived  within  the  knowledgebase, 
relationships  will  be  defined  between 
behaviors  and  functional  capabilities  to 
support  analysis  and  trade  off  of  specific 
deployments.  As  depicted  in  Figure  3, 
candidate  deployments  can  be  evaluated 
early  in  the  analysis  process  by  modeling 
deployments  from  the  knowledgebase  in 


Figure  3:  HRI  Modeling  Methodology 


an  intelligent  systems  behavior  simulation.  For  this  step,  tasks  are  extracted  from  the  knowledgebase  and  allocated  to 
specific  hardware  and  crewmember  agent  models,  with  the  behavior  simulator  executing  the  deployment  as  a  set  of 
interacting  agents.  External  inputs  to  the  behavior  simulation;  representing  platform  subsystems,  other  vehicles,  and 
dismounted  soldiers  is  provided  via  constructive  simulation  tools.  Once  a  deployment  has  been  evaluated,  portions  of 
the  knowledgebase  may  be  provided  to  component  vendors,  representing  a  set  of  executable  requirements  to  include  the 
logical  data  model  and  interactions  required  with  other  subsystem  components.  As  components  are  acquired,  they  are 
evolved  and  tested  in  an  iterative  fashion  through  various  program  phases  to  include  system  integration  testing,  and 
field  experimentation.  Results  from  each  of  these  phases  are  fed  back  into  the  knowledgebase  as  applicable.  The 
capturing  of  behaviors  within  the  ontology  is  further  defined  within  section  4.2.2  and  the  modeling  approach  to  support 
deployment  development  and  analysis  is  further  defined  in  section  4.2.1.  In  parallel,  technologies  will  be  explored  (see 
section  5)  that  will  optimize  the  soldier’s  performance  for  given  missions.  For  example,  due  to  the  fact  that  a 
dismounted  soldier  will  have  a  reduced  amount  of  display  space  as  compared  to  a  mounted  counterpart,  he/she  might 


utilize  technologies  such  as  speech  recognition  or  gesturing  as  an  alternative  input  device  to  interface  with  the  system. 
The  end  goal  is  to  put  the  right  equipment  into  the  soldier’s  hands  that  best  fit  his/her  overall  mission  requirements. 


4.  PROGRAM  COMPONENTS 

Four  key  components  serve  as  the  foundation  for  the  HRI  ATO  systems  engineering  approach  to  human-robot 
interactions:  Architecture,  modeling  and  simulation,  interface  scalability,  and  intelligent  agents/adaptive  automation. 
These  items  were  provided  to  PM-UA  as  part  of  a  Technology  Transition  Agreement^  that  was  delivered  in  November 
of  2004  and  will  refine  deliverables  through  FY08.  Each  component  is  detailed  in  the  sections  that  follow,  including 
applicable  technologies  and  methodologies  under  development/refmement  within  the  DoD  and  commercial  world  that 
have  bearing  on  HRI  ATO  program  requirements. 

4.1.  Architecture 

In  order  to  ensure  the  integration  of  HRI  components  and  HRI  product  compatibility  to  Army  weapon  systems,  the  HRI 
program  will  define  a  reference  architecture,  supporting  product  definition,  development,  integration,  and  test.  The 
reference  architecture  will  be  defined  utilizing  a  systems  engineering  process  to  analyze  and  capture  HRI  system 
requirements,  future  force  Army  mounted  and  dismounted  system  requirements.  Army  force  operational  requirements 
(supporting  derivation  of  vehicle  tasks  and  vehicle/crew  behaviors).  Army  technical  architectural  requirements,  and 
intelligent  system  architecture  discipline  requirements. 

HRI  system  architecture  requirements  will  be  derived  from  a  varying  set  of  sources  to  include: 

•  HRI  target  platform  requirements  with  respect  to  scalable  interface  configurations  and  platforms  defining 
scalability  and  extensibility; 

•  HRI  technology  (e.g.,  crew  control  and  operator  interface  technologies,  decision  aiding,  sensors,  common 
Warfighter-Machine  Interface  (WMI))  requirements  compiled  via  program  and  technology  assessments; 

•  HRI  program  transition  requirements  supporting  the  future  force  to  include  FCS,  Future  Force  Warrior  (FFW) 
and  HRI  technology  spiral  integration  into  Stryker  and  current  force  systems;  and 

•  Force  structure  system  requirements,  defining  the  tactical  operational  behavior  of  the  robotic  and  host 
platforms  within  platoon  configurations;  and  Joint  architecture  requirements  as  they  relate  to  architecture 
standards  and  interfaces,  defined  within  the  Joint  Technical  Architecture  and  Joint  Technical  Architecture  - 
Army. 

Following  the  establishment  of  the  HRI  system  architecture  requirements,  a  reference  system  architecture  design  will  be 
initiated  to  establish  a  logical  model  of  architecture  components,  attributes,  and  interfaces  to  facilitate  the  creation  and 
integration  of  system,  subsystem,  and  component  models  supporting  HRI  virtual  and  physical  integration.  Primary 
inputs  to  the  HRI  system  architecture  design  include  the  review  and  synthesis  of:  Army  weapon  system  architecture 
initiatives  to  include  the  Joint  Technical  Architecture  and  Joint  Technical  Architecture  -  Army,  the  Weapon  System 
Common  Operating  Environment  and  the  FCS  System  of  System  Common  Operating  Environment; 

TARDEC  Intelligent  Systems  architecture  initiatives,  to  include  the  Vetronics  Reference  Architecture,  embedded 
simulation  architecture,  Vetronics  Technology  Integration  crewstation  and  system  architecture,  WMI  guidelines  and 
common  WMI  specifications;  and  the  TARDEC  Intelligent  Ground  Vehicle  (IGV)  taxonomy  (see  section  4.2. 2. 3).  The 
system  architecture  design  will  utilize  the  IGV  taxonomy  as  a  basis  to  capture  and  decompose  vehicle  tasks  to  tactical 
behaviors  and  subsystem/system  components  that  execute  those  behaviors.  In  this  context  the  term  “componenf’ 
implies  subsystems,  architecture,  intelligent  agents,  or  scalable  interface  operators.  Components  will  be  identified  in 
accordance  with  the  HRI  architecture  requirements  and  a  functional  classification  and  decomposition  approach  will  be 
employed  to  derive  a  reference  architecture  model  identifying  components,  component  interfaces,  component  attributes, 
and  component  relationships  and  dependencies.  Component  attributes  will  be  defined  to  not  only  capture  operational 
characteristics,  but  to  capture  architecture  physical  and  usability  aspects  as  well  (e.g.,  weight,  cost,  volume,  lifespan, 
mean  time  between  failure). 

The  reference  architecture  will  be  specified  using  the  Unified  Modeling  Language  (UML)  and  Ontology  Web  Language 
(OWL)  and  will  capture  system-wide  design  decisions,  architecture  context,  system  components,  concept  of  execution, 
and  interface  design. 


4.2.  Modeling  and  Recursive  Simulation 

The  following  section  describes  various  HRI  ATO  modeling  and  simulation  methodologies  and  tools  supporting 
component,  constructive,  and  virtual  modeling  and  simulation  to  be  applied  in  the  definition  and  development  of  HRI 
subsystems  and  systems. 

4.2.1.  End  to  End  Modeling  Approach 

The  HRI  ATO  will  employ  a  model-based  development  process  facilitating  the  initial  realization  of  system/subsystems 
as  computer  models  that  are  iteratively  refined  and  validated  through  simulated  tests.  For  each  iteration,  the  fidelity  of 
the  model  and  the  simulated  test  will  be  increased  until  it  can  be  demonstrated  that  the  modeled  design  meets  all  of  the 
system  requirements  for  the  intended  environment.  At  that  point  a  physical  system  can  be  realized  from  the  model 
definition  in  a  laboratory  prototype  form.  The  physically  realized  prototype  system  can  be  iteratively  tested  in  the 
simulated  environment  until  it  demonstrates  the  ability  to  meet  all  allocated  system  requirements,  at  which  point  the 
system  can  be  realized  in  a  form  suitable  for  its  intended  environment  and  tested  in  the  field  though  user 
experimentation.  This  process,  currently  being  matured  within  TARDEC  is  known  as  the  end  to  end  modeling  process. 

The  end  to  end  modeling  process  maps  to  the  general  systems  engineering  design  process,  aligning  methodologies  and 
tools  to  assist  in  early  and  frequent  up  front  modeling  in  the  requirements  definition  process  through  preliminary  and 
detailed  design.  The  modeling  through  these  phases  of  a  system’s  lifecycle  will  be  iterative  and  it  is  highly  desirable  to 
select  tools  and  methodologies  that  facilitate  a  continuous  transition  as  the  models  increase  in  fidelity.  It  is  envisioned 
that  modeling  phases  within  the  end  to  end  modeling  process  will  produce  correlated  constructive,  component,  virtual, 
and  physical  models  to  include  scalable  interface  models,  host  vehicle  models  into  which  scalable  interfaces  will  be 
integrated,  and  robotic  vehicle  models  that  will  be  controlled  by  scalable  interfaces  that  control  robotic  assets,  whether 
they  are  configured  within  a  vehicle  crewstation,  dismounted  handheld  device,  or  mixed.  Models  developed  early  on  in 
the  requirements  and  design  phase  will  be  archived  such  they  may  serve  as  a  reusable  model  base  for  programs  with 
similar  requirements. 

The  HRI  program  is  currently  defining  the  phases  of  the  end  to  end  modeling  approach  to  be  employed  within  the 
program,  the  relevant  tools  and  methodologies,  and  the  desired  mapping  of  the  end  to  end  modeling  approach  to  the 
systems  engineering  design  process.  Some  of  the  tools  and  methodologies  under  investigation  are  described  in 
subsequent  sections  of  this  report. 

4.2.2.  Constructive  Modeling 

A  number  of  constructive  modeling  tools  and  methodologies  will  be  examined  for  inclusion  in  the  HRI  ATO.  This  will 
include  stochastic  tools  such  as  CASTFOREM  and  COMBAT  XXI,  component  models  developed  by  MathWorks, 
MATLAB,  SimuLink,  and  AutoCAD,  behavior  modeling  tools  such  as  OWL-S,  cognitive  modeling  tools  like  ACT-R, 
IMPRINT  and  SOAR. 

Stochastic  modeling  tools  like  CASTFORUM  and  Combat  XXI  will  be  employed  within  the  end  to  end  modeling 
process  to  model  initial  vehicle  level  requirements  with  respect  to  required  system  capabilities.  Once  established,  these 
requirements  will  be  flowed  through  the  end  to  end  modeling  process  to  support  the  derivation  of  human  robotic 
interfaces  and  evaluate  human  robotic  interface  deployments. 

The  HRI  program  will  develop  stand  alone  and  integrated  constructive  component  models,  representative  of  common 
scalable  interface  architecture  system/subsystem  and  application  components.  Component  models  used  to  support 
virtual  integration  and  test  will  focus  on  analysis,  and  support  dynamic  simulation  of  key  integration  parameters  such  as 
power  loading,  data  bus  loading,  processor/memory  loading,  thermal  loading,  space  claim,  interface  compatibility,  and 
operator  workload.  Embedded  simulation  models  will  be  developed  to  focus  on  operational  performance  analysis  and 
will  support  dynamic  simulation  of  component  operation  in  a  distributed  simulation  environment.  These  models  will 
also  have  access  to  simulation  scenario  and  ground  truth  information  in  order  that  they  can  simulate  the  operational 
behavior  of  complex  automated  systems.  In  addition  to  scalable  interface  models,  the  HRI  program  is  also  developing 
Manned  Ground  Vehicle  (MGV),  Unmanned  Ground  Vehicle  (UGV)  and  Unmanned  Air  Vehicle  (UAV)  component 
models  to  support  simulation,  integration,  and  validation  of  scalable  interface  concepts  prior  to  build  and  field  test. 


The  HRI  ATO  program  is  defining,  employing,  and  refining  an  IGV  ontological  modeling  technique  to  specify  systems 
of  systems  behavioral  views  for  both  manned  and  unmanned  ground  vehicles.  The  IGV  ontology  models  a  group  of 
vehicles  (e.g.,  a  Unit  of  Action)  as  a  hierarchy  of  interconnected  intelligent  agents  that  spans  each  of  the  command 
echelons  (i.e.,  company  commander,  platoon  leader,  section  leader)  as  well  as  the  vehicle  architecture  (i.e.,  major, 
vehicle  systems,  subsystems,  and  components).  The  behavior  of  each  agent  is  specified  as  a  set  of  services  defined 
using  the  OWL-S  web  service  ontology  developed  by  DARPA.  OWL-S  also  provides  the  language  to  map  services  to 
intelligent  agents  and  describe  interactions  between  services.  Although  work  has  already  started  on  the  development  of 
the  IGV  ontology  many  more  missions  must  be  modeled  to  support  the  HRI  project.  A  functional  decomposition  of 
anticipated  missions  of  UGV  and  UAV  equipped  units  will  be  conducted  using  the  4D/RCS  methodology  to  identify  the 
essential  tactical  behaviors  of  each  vehicle  (manned  or  unmanned)  in  the  unit  and  the  interactions  between  each  vehicle 
in  the  unit.  This  decomposition  will  begin  with  the  analysis  of  Army  reference  materials  defining  missions,  tactics, 
techniques  and  procedures.  The  functional  decomposition  will  also  include  the  identification  of  the  tasks  that  must  be 
conducted  within  each  vehicle  to  control  each  level  of  equipment  to  accomplish  the  tactical  behaviors.  The  IGV 
behavior  models  will  be  developed  using  open  source  tools  such  as  Protege-2000.  As  HRI  technology  concepts  mature, 
IGV  behaviors  will  be  allocated  to  human  operators  or  computer  automation  in  separate  IGV  deployment  models  to 
reflect  candidate  system  design  concepts.  Representations  of  vehicles  within  the  ontological  model  will  be  kept 
consistent  with  an  HRI  reference  architecture  model  to  support  the  allocation  of  tasks  to  agents  that  correspond  to 
physical  entities  in  the  HRI  architecture.  IGV  behavioral  models  will  serve  as  a  knowledge  base  for  tactical  behaviors 
which  will  also  be  used  as  a  reference  sources  for  the  design  of  subsystem  models,  constructive  operators,  or  decision 
aiding  approaches  such  as  intelligent  agents. 

The  HRI  program  will  provide  a  means  to  model  the  cognitive  behavior  required  to  conduct  ground  combat  vehicle 
operations.  These  models  will  be  developed  to  capture  the  intelligence  required  to  control  and  interact  with  robotic 
systems.  This  modeling  activity  will  support  simulation  integration  to  analyze  automation  requirements  and  to  trade-off 
deployments/conceptualizations  prior  to  physical  hardware  design  and  development.  Three  cognitive  modeling 
technologies  currently  under  investigation  and/or  development  within  TARDEC  to  support  the  HRI  program  are  ACT- 
R,  IMPRINT,  and  SOAR. 

Adaptive  Character  of  Thought  -  Rational  (ACT-R)  specifies  a  cognitive  architecture  that  can  be  utilized  to  model  a 
wide  range  of  human  cognition  and  is  targeted  at  the  atomic  level  of  thought  (cognitive,  perceptual,  and  motor).  For  the 
HRI  program,  TARDEC  is  analyzing  the  benefits  and  potential  of  employing  the  ACT-R  modeling  technique  to 
characterize  cognitive  behavior  and  is  also  analyzing  data  resulting  from  the  IMPRINT/ACT-R  integration  experiments 
conducted  by  the  ARL-HRED. 

Improved  Performance  Research  Integration  Tool  (IMPRINT)  models  are  used  to  constructively  simulate  cognitive 
behaviors.  IMPRINT  is  dynamic,  stochastic  discrete  event  network  modeling  tool  that  assesses  the  interaction  of 
soldier  and  system  performance  throughout  the  system  lifecycle.  In  order  to  support  virtual  test  of  scalable  interface 
concepts,  the  HRI  program  will  define  a  set  of  IMPRINT  models,  to  include  tasks  identified  in  the  MGV  behavioral 
models.  These  IMPRINT  models  will  include  predefined  external  stimuli  to  enable  the  simulation  of  the  model  in 
predefined  HRI  test  vignettes.  Initial  models  will  be  modified  to  define  the  specific  allocation  of  tasks  to 
operators/automation;  representing  the  specific  interactions  between  the  HRI  scalable  interface  and  operator  in 
accordance  with  the  specific  WMI  technologies  defined  in  the  scalable  interface  concept.  Multiple  versions  of  these 
IMPRINT  models  can  be  created  with  the  task  execution  times  and  workload  adjusted  to  reflect  specific  applications  of 
HRI  technologies.  Each  version  can  be  evaluated  for  a  set  of  test  scenarios  and  the  results  compared  to  assist  in  the 
selection  of  the  most  promising  HRI  configuration.  As  new  IMPRINT  models  are  developed,  they  will  be  archived 
within  a  model  repository  for  possible  reuse  by  the  Government  or  other  scalable  interface  contractors. 

State  Operator  and  Result  (Soar)  is  a  general  cognitive  architecture,  developed  by  the  University  of  Michigan  that  has 
been  employed  in  the  development  of  systems  that  exhibit  intelligent  behavior  across  a  wide  array  of  domains  to  include 
artificial  intelligence  and  cognitive  science  Soar  addresses  two  primary  aspects  of  cognitive  architecture:  1)  a  fixed  set 
of  mechanisms  and  structures  that  process  content  to  produce  behavior  and  2)  the  theory,  or  point  of  view,  about  what 
cognitive  behaviors  have  in  common.  For  the  HRI  program,  TARDEC  is  analyzing  the  application  of  the  Soar 
architecture  for  cognitive  modeling  and  for  translation  of  cognitive  behaviors  to  decision  aiding  agents  to  control 
autonomous  unmanned  assets. 


4.2.3.  Vehicle  Performance  Modeling 

For  the  past  several  years,  TARDEC  has  been  working  towards  the  evolution  of  a  constructive  modeling  technique  to 
represent  and  integrate  human  operators  within  high  fidelity  M&S  environments.  This  technique  to  define  high- 
resolution  constructive  models  of  a  ground  combat  vehicle  and  its  crew  has  been  termed  Vehicle  Level  Human 
Performance  Modeling  (VLHPM).  The  VLHPM  supports  the  prediction  of  crewmember  performance  given  a  specific 
vehicle  system  configuration  (e.g.,  specific  mobility,  lethality,  surveillance,  and  survivability  subsystems)  and  a 
specific  set  of  warfighter-machine  interface  technologies  (e.g.,  multifunction  displays,  voice  control,  heads  up  displays). 
The  VLHPM  is  capable  of  operating  in  a  distributed  simulation  environment  using  HLA  protocols  to  enable  the  creation 
of  complex  battlefield  environments  to  fully  exercise  the  crew  models.  The  VLHPM  supports  optimization  of  operator 
task  allocations  and  warfighter-machine  interface  designs  prior  to  development  of  more  expensive  man-in-the-loop  crew 
station  simulators.  The  architecture  of  the  VLHPM  is  comprised  of  a  vehicle  model  integrated  to  a  human  performance 
model. 

The  vehicle  model  is  implemented  utilizing  the  TARDEC  Embedded  Simulation  System  (ESS),  which  is  a 
configurable,  high-resolution  simulator  of  ECS  combat  vehicle  organizations  designed  to  support  constructive,  and 
human-in-the-loop  (HITL)  real-time  (wall-clock)  mock  battle  simulation  experiments.  The  ESS  provides  a  configurable 
vehicle  model,  which  in  turn  provides  detailed  simulations  of  subsystems  such  as  mobility,  lethality,  survivability, 
sensing,  and  communications.  It  can  be  employed  to  model  and  simulate  either  manned  or  unmanned  ground  and  air 
vehicles,  has  Semi  Automated  Forces  (SAF)  capability,  and  is  DIS/HLA  compliant.  The  ESS  models  the  latest  PCS 
concept  vehicle  designs  (MGV,  UGV,  UAV  and  Unattended  Ground  Sensors  (UGS))  and  will  be  altered  to 
accommodate  program  needs. 

4.2.4.  Virtual  System  Integration  Laboratory  (VSIL) 

The  TARDEC  VSIL  is  a  constructive  simulation  environment  designed  to  facilitate  vetronics  systems  engineering  of 
intelligent  ground  vehicle  systems.  The  VSIL  defines  a  logical  model  with  an  associated  configurable  simulation  object 
base  that  can  be  employed  within  a  constructive  simulation  at  various  levels  of  fidelity  to  rationalize  and  evaluate 
system/subsystem  deployments  prior  to  final  allocation  and  hardware  acquisition.  VSIL  deployments  support  analysis, 
allocation,  and  tradeoff  within  key  areas  of  the  intelligent  ground  vehicle  domain  to  include:  Vetronics  (data  control 
and  distribution,  computing/knowledge  resources,  controls  and  displays,  power  management  and  distribution), 
intelligent  agents  (human/machine),  physical  allocation  (power,  weight,  volume,  thermal,  and  other  environmental), 
workload  allocation  (human/machine),  and  planning  and  preparation. 

The  three  primary  VSIL  interfaces  are  the  repository  interface;  the  simulation  execution  environment  interface;  and  the 
performance,  analysis,  and  measurement  interface.  Through  these  interfaces,  VSIL  model  developers  and  VSIL  system 
deployment  developers  can  create,  execute,  analyze,  and  archive  VSIL  system  models.  The  repository  interface 
provides  for  the  import/export  of  repository  models  and  the  configuration  of  models  into  system/subsystem 
deployments.  In  addition,  this  interface  can  be  utilized  to  create  test  scenarios  that  define  external  stimuli  representing 
battlefield  events.  The  simulation  execution  environment  interface  governs  simulation  startup/initialization,  shutdown, 
and  execution  control.  Finally,  the  performance,  analysis,  and  measurement  interface  provides  a  means  to  analyze 
simulation  performance  through  the  measurement  of  system  measures  of  performance  and  measures  of  effectiveness 
based  on  attributes  defined  in  the  VSIL  reference  model  architecture. 

Within  the  HRI  ATO  program,  the  VSIL  will  be  used  to  evaluate  specific  deployments  of  scalable  interface  components 
during  the  preliminary  and  detailed  design  of  a  scalable  interface.  Each  evaluation  will  provide  data  on  the 
compatibility  of  the  selected  scalable  interface  components  with  each  other  and  with  the  vetronics  systems  defined  for 
the  host  vehicle.  The  evaluation  will  identify  vetronics  resource  shortfalls  that  could  occur  at  any  time  during  a  mission 
(e.g.,  insufficient  battery  power  to  sustain  a  remotely  deployed  scalable  interface  during  extend  periods  of  silent  watch). 
The  evaluation  will  also  provide  an  insight  to  the  operator  work  load  during  different  phases  of  system  operation.  Use 
of  the  VSIL  facilitates  the  conduction  of  trade-offs  to  refine  deployment  concepts  and  as  deployment  concepts  mature 
from  preliminary  design  through  detailed  design,  component  models  with  increasing  levels  of  fidelity  can  be  used  to 
support  detailed  analysis  in  accordance  with  the  increasing  levels  of  fidelity  in  the  design  definition. 


4.2.5.  Physical  Simulation 


Physical  simulation  will  be  utilized  within  the  HRI  program  as  a  step  within  the  end  to  end  modeling  process  to  validate 
operator  HMI  performance  prior  to  vehicle  field  testing.  Physical  simulation  is  conducted  within  the  TARDEC  Ground 
Vehicle  Simulation  Facility  by  integrating  the  scalable  interface  and  operator  into  a  motion  based  simulator  and 
executing  tactical  operational  scenarios  requiring  operator  interaction.  Operator  performance  and  condition  will  be 
evaluated  to  validate  the  effectiveness  of  the  scalable  interface  under  test.  Deficiencies  will  be  documented  and 
addressed  via  redesign  at  the  appropriate  predecessor  phase  within  the  end  to  end  modeling  cycle. 

The  HRI  program  is  coordinating  with  the  High-Fidelity  Ground  Platform  &  Terrain  Mechanics  Modeling  ATO,  a 
collaborative  effort  among  TARDEC,  ERDC  and  ARE,  to  leverage  their  high  fidelity  models  of  FCS  ground  platforms 
and  their  corresponding  terrain  to  conduct  assessments  of  mobility,  durability,  mission  module,  and  moving  vehicle 
operations. 


4.3.  Decision  Aiding 

While  the  concept  of  decision  aiding  has  been  in  existence  for  several  years,  the  manner  in  which  decision  aids  are  best 
suited  to  support  crew  operators  and  the  manner  in  which  decision  aiding  architectures  should  be  developed  to  support 
ease  of  integration  for  disparate  vendor  solutions  and  technology  has  not  been  adequately  addressed  (Fig  4). 

Areas  that  need  to  be  further  explored 
and  matured  include: 

•  Fevels  of  autonomy. 

Typically  levels  of 
autonomy  will  minimally 
include  no  autonomy 

(decision  aide  may  be  aware 
such  that  it  could  react  to  a 
change  in  state,  but  is 
completely  passive),  semi- 
autonomous  (decision  aide  is 
in  the  loop  and  performs 
some  functionality,  operator 
provides  final  authority  for 
any  mission  critical 

function),  and  fully- 

autonomous  (decision  aide  is  fully  engaged  and  performing  tasks  typically  assigned  to  a  crewmember). 

Decision  aiding  architecture.  Both  the  FCS  program  and  science  and  technology  programs  have  been 
experimenting  with  designing  decision  aiding  architectures.  In  order  to  be  successful,  a  decision  aiding 
architecture  will  require  the  ability  to  adapt  over  time  and  architecture  terms,  behaviors,  and  components  need 
to  be  defined  that  will  facilitate  the  ability  to  integrate  advancements  in  decision  aide  technologies  over  time. 
One  key  area  is  the  definition  of  the  system  knowledge  base  and  the  manner  in  which  decision  aids,  planners, 
and  system  components  will  interface  to  enhance  and  utilize  elements  of  the  knowledge  base. 

Commercial  infrastructure.  Commercial  knowledge  architecture  infrastructure  is  rapidly  developing  to  include 
knowledge  management,  semantic  networks,  intelligent  agents,  and  web  based  service  architectures.  The  FCS 
program  has  been  analyzing  the  manner  in  which  commercial  tools,  techniques,  and  paradigms  might  best  map 
to  support  system  of  system  agent  relationships  and  interactions.  As  methodologies  are  defined  to  decompose 
tasks  and  behaviors,  tools  need  to  be  defined  to  capture  operational  and  system  level  architecture  data  and 
facilitate  the  analysis,  automation,  modeling  and  simulation,  and  eventual  development  and  integration  of 
intelligent  agents.  Commercial  tools  in  the  areas  of  ontology,  modeling  and  simulation,  and  web  services  need 
to  be  analyzed  against  these  requirements. 

Extended  application.  Following  the  establishment  of  decision  aiding  architecture,  extended  applications  need 
to  be  analyzed  and  developed  to  determine  the  best  way  in  which  decision  aids  can  support  improvements  in 
the  operator’s  ability  to  perform  his  mission  with  respect  to  human  robotic  interaction  and  teaming.  One 
example  area  is  in  task  sharing/shedding,  which  would  combine  sensors  integrated  to  the  decision  aiding 


system  to  monitor  crew  condition  and/or  loading  and  to  adapt  to  either  take  on  priority  tasks  or  share  the 
workload  (i.e.,  perform  as  an  autonomous  interactive  crewmember). 

4.4.  Common  WMI 

This  section  addresses  activities  under  investigation  within  the  HRI  ATO  to  standardize  WMI  and  WMI  components 
within  the  target  platform  scalable  interface  domain.  The  approach  under  investigation  addresses  the  architectural  and 
requirements  basis  for  WMI  commonality  such  that  a  common  WMI  can  be  defined  that  is  not  only  configurable  and 
extensible,  but  is  also  reconfigurable  and  adaptable  to  support  system  upgradeability  and  interchangeability  of  robotic 
assets  and  mission  packages. 

4.4.1.  WMI  Guidelines 

A  primary  base  component,  supporting  the  standardization  of  a  common  WMI  is  the  establishment  of  a  set  of  guidelines 
governing  WMI  methodology  and  principles  within  the  domain.  The  HRI  ATO  will  analyze  requirements  to  develop  a 
consistent  set  of  guidelines  that  can  be  interpreted  within  an  interface  to  support  common  look,  feel,  and  scalability. 
The  HMI  guidelines  will  address  the  following  areas  of  scalable  interface  design:  configurability,  display  layout,  data 
logging,  operator  login,  status,  and  modes  of  operation.  Configurability  addresses  dynamic  reconfigurability  of 
controller  and  robot  interfaces  providing  the  ability  to  upload  platform  configuration  specific  data  for  display.  Platform 
specific  data  could  include  mission  package  configuration,  mission  package  controls  and  status,  as  well  as  robotic 
platform  specific  controls  and  status.  Display  layout  addresses  the  manner  in  which  a  screen  can  be  physically  and/or 
virtually  addressed  supporting  the  placement  of  controls  and  status  indicators.  General  terms  and  methods  applicable 
across  interfaces  would  be  defined  in  an  abstract  manner  such  that  they  could  be  mapped  accordingly  within  a  physical 
interface  design,  for  example  warning,  caution,  and  alert  (WCA)  levels  and  help.  To  the  degree  applicable,  static 
display  areas  will  be  defined  for  the  display  of  critical  data  such  as  status  and  WCA’s.  Data  logging  addresses  the  type 
of  data  that  may  be  logged  for  training,  validation,  and/or  analysis  purposes.  Logged  data  could  include  WCA,  audio, 
images,  and  session  data.  Operator  login  addresses  the  manner  in  which  an  operator  gains  access  to  the  WMI,  systems 
controlled  by  the  interface,  and  the  role  access  of  the  operator  within  a  session.  Status  addresses  the  manner  in  which 
operator  critical  status  information  is  defined  and  accessed.  Critical  status  information  could  include  robotic  vehicle 
status,  mission  status,  vehicle  orientation,  communication  link  status,  and  battery  levels.  Modes  of  operation  address 
controller  modes  of  operation  and  the  interface  behavior  within  each  respective  mode.  Example  modes  could  include 
setup  (configuration,  diagnostics,  log/other  status),  pre/post  operational  (mission  planning  and  mission  analysis), 
operational  (tactical  operations),  sleep  (reserve),  and  training  (embedded  training,  mission  playback  and  analysis). 

4.4.2.  Description  Techniques 

Description  techniques  provide  a  mechanism  to  define  scalable  interface  graphical  objects  and  the  manner  in  which 
these  objects  are  defined  for  interchange,  display,  and  behavior  within  a  system.  For  the  HRI  program,  description 
techniques  are  being  analyzed  such  that  a  device/graphics  independent  hierarchical  WMI  object  classification  can  be 
defined  where  lower  level  (detailed)  objects  can  be  specified  to  inherit  and  extend  parent  attributes.  For  each  graphical 
object,  behavior  (attributes,  controls,  and  status),  graphical  realization  (physical  and  graphical  configurations), 
application  interface  (methods  and  effects),  and  interchange  (XML  specification  of  representation/classification)  will  be 
defined.  Within  the  HRI  program  these  graphical  classifications  will  be  utilized  to  specify  and  interchange  human- 
machine  interface  between  robotic  vehicles  and  controllers  supporting  discover  and  reconfiguration  of  robotic  systems. 
In  addition,  the  employment  of  description  techniques  would  facilitate  role  based  scalable  interface  development  in  that 
a  controller  targeted  to  serve  a  specific  purpose;  e.g.,  diagnostics  or  palette  loading/unloading  would  process  only  the 
essential  XML  tag  controls  required  for  the  mission  at  hand,  while  ignoring  non-essential  XML  tag  controls. 

4.4.3.  WMI  Scalability 

The  HRI  program  is  investigating  various  methods  to  address  WMI  scalability  to  include  graphical/physical  scalability 
as  well  as  configuration  and  prioritization  of  interface  information.  With  respect  to  graphical/physical  scalability, 
alternate  representations  or  scaling  of  graphical  controls  could  be  defmed/standardized  within  the  description  technique 
(e.g.,  graphical  control  visualization  for  a  large  scale,  medium  scale,  or  small  scale  display).  With  respect  to 
configuration  and  prioritization  of  scalable  interface  information,  techniques  are  being  analyzed  to  encode  information 


priority  within  XML  schemas  such  that  a  WMI  can  display  information  in  priority  order  in  accordance  with  available 
display  area. 

4.4.4.  WMI  Portability 

The  HRI  program  is  analyzing  methods  to  port  WMLs  across  platforms  to  include  the  analysis  of  description  techniques 
that  define  WMLs  in  a  device,  graphical,  and  operating  system  independent  manner.  Portability  analysis  will  also 
investigate  graphical  interchange  techniques  and  device/programming  language  independent  representation  of  graphical 
control  methods. 


5.  TECHNOLOGIES  UNDER  INVESTIGATION 

In  addition  to  the  methodologies  employed  above  for  executing  the  program,  the  HRI  ATO  will  explore  various  current 
and  emerging  technologies  as  well  as  commercial  operator  device  interfacing  techniques  for  potential  inclusion.  Many 
of  these  are  currently  being  investigated  with  respect  to  the  level  of  applicability  and  the  manner  in  which  they  would 
facilitate  operator  workload  in  either  a  positive  or  negative  regard. 

5.1.  Head  Mounted  Display  (HMD) 

An  HMD  is  a  headset  used  mostly  with  virtual  reality  systems.  An  HMD  can  be  a  monocle,  a  pair  of  goggles,  or  a  full 
helmet.  In  front  of  each  eye  is  a  tiny  monitor.  Some  project  the  data  to  a  piece  of  glass,  allowing  the  data  to  be  overlaid 
to  the  real  world  view.  In  addition,  most  HMD’s  include  a  head  tracker  so  that  the  system  can  respond  to  head 
movements. 

5.2.  Head  Tracker 

A  head  tracker  is  a  device  used  to  detect  the  movement  and  position  of  the  users  head.  There  are  five  types  of  head 
trackers  in  use  today:  mechanical,  optical,  magnetic,  acoustic,  and  inertial.  Each  uses  a  different  approach  to  collecting 
the  head  position  data,  and  each  has  advantages  and  disadvantages  that  must  be  weighed  against  specific  mission 
requirements. 

5.3.  Displays 

Most  WMI  displays  in  currently  fielded  ground  combat  vehicles  are  based  on  Active  Matrix  Liquid  Crystal  Display 
(AMLCD)  color  flat  panel  technology.  These  devices  can  display  at  least  512  unique  colors.  This  is  a  requirement  in 
order  to  display  color  maps  and  enhance  the  effectiveness  of  the  user.  Typically  these  currently  fielded  WMI  displays 
are  commercial  or  industrial  quality  from  offshore  sources  that  have  been  ruggedized  by  a  3rd  party  or  vehicle  system 
integrator.  Ruggedization  might  include  enhanced  packaging  for  high  shock  and  vibration  environments,  internal 
heaters  to  extend  low  temperature  operation,  added  protection  for  the  display  glass  itself,  operation  from  28V  vehicle 
power,  and/or  the  replacement  of  commercial  connectors  with  higher  reliability  military  equivalents.  The  WMI  displays 
themselves  are  typically  augmented  with  one  or  more  operator  input  device  such  as  a  touch  screen,  pointing  device,  or 
bezel  mounted  switches. 

Recently  the  trend  has  been  to  migrate  toward  an  “all  digital”  interface  between  the  display  and  the  computer  system. 
Early  systems  used  an  analog  component  type  interface.  This  signal  had  to  be  digitized  at  the  display  since  an  AMLCD 
is  inherently  a  digital  device.  An  “all  digital”  interface  improves  display  quality  and  supports  higher  display  resolutions 
with  more  colors.  There  have  been  issues  with  display  obsolescence  and  maintaining  adequate  supply  for  production 
and  spares.  The  commercial  AMLCD  market  is  primarily  driven  by  Laptop  computers  since  they  consume  most  of  the 
displays.  If  a  certain  size  or  resolution  display  that  is  used  on  a  military  vehicle  falls  out  of  favor  commercially,  then 
supply  problems  for  the  military  can  arise.  This  fact  is  leading  a  push  to  choose  a  new  technology  to  replace  the  current 
AMLCD  displays.  There  has  been  a  push  in  flexible  display  technology  that  may  provide  unique  solutions  for 
dismounted  soldiers,  and  will  be  monitored. 

5.4.  3D  Audio 

A  three-dimensional  audio  (3D  Audio)  system  allows  for  the  placement  of  sound  cues  in  three-dimensional  space. 
These  sounds  are  produced  using  traditional  stereo  speakers.  This  is  accomplished  by  dynamically  analyzing  the  sound 


coming  from  the  speakers  and  sending  feedback  to  the  sound  system  so  that  it  can  readjust  the  sound.  Sounds  are 
usually  placed  using  a  graphical  user  interface  (GUI)  provided  with  the  3D  Audio  system.  Those  systems  that  do  not 
provide  a  GUI  often  provide  an  Application  Programmer’s  Interface  (API)  to  utilize  the  3D  Audio  capabilities.  Once  a 
sound  is  placed  in  the  3D  world,  its  position  is  produced  based  on  the  user’s  orientation  and  head-related  transfer 
function  (HRTF).  Head  trackers  are  used  by  most  3D  Audio  systems  to  obtain  user  orientation  information.  These  head 
trackers  are  commonly  linked  to  3D  Audio  systems  using  serial,  USB,  or  Ethernet  connections.  HRTF’s  are  generated 
based  on  the  user’s  head  size.  The  number  of  HRTF’s  provided  with  a  3D  audio  system  varies.  Some  systems  provide 
a  way  to  generate  new  HRTF’s  using  additional  hardware  or  make  different  HRTF’s  available  via  the  company’s 
website. 

5.5.  Speech  Recognition 

Speech  Recognition  deals  with  designing  computer  systems  that  can  recognize  spoken  words.  Note  that  speech 
recognition  implies  only  that  the  computer  can  take  dictation,  not  that  it  understands  what  is  being  said.  The  most 
powerful  systems  can  recognize  thousands  of  words;  however,  they  generally  require  an  extended  training  session 
during  which  the  computer  system  becomes  accustomed  to  a  particular  voice  and  accent.  Speech  recognition  can  aid  in 
simple  tasks  such  as  data  entry  and  menu  navigation.  More  complicated  tasks  such  as  directly  controlling  one  or  more 
remote  assets  can  be  accomplished,  but  the  effectiveness  of  this  application  has  yet  to  be  determined.  Voice  quality 
may  change  under  adverse  conditions  with  high  noise  and  vibrations.  This  will  cause  the  accuracy  of  the  speech 
recognition  system  to  degrade.  Adding  Active  Noise  Reduction  (ANR)  to  the  speech  recognition  system  would  be  one 
way  to  counteract  this  condition. 

5.6.  Controls 

Most  scalable  interfaces  are  targeted  to  various  platform  hardware  configurations,  largely  due  to  vendor  specific 
requirements  as  well  as  user  community  and/or  program  end  application  requirements.  Fielded  scalable  interfaces  tend 
to  be  packaged  as  a  display  and  associated  physical  input  controls  (e.g.,  switches,  dials,  knobs,  and  joysticks).  The 
displays  may  provide  for  user  input  via  a  touch  screen,  allowing  for  graphical  user  input  as  well.  The  use  of  control 
devices  are  generally  based  on  the  capability  it  will  provide.  Axis  devices,  such  as  levers,  pedals,  and  joysticks,  are 
predominately  used  to  control  movement  tasks.  Buttons  and  switches  can  also  be  used  to  control  mobility,  but  may  not 
provide  the  sensitivity  needed  for  fine  control.  Buttons  and  switches,  whether  physical  or  graphical,  are  more  often 
used  to  control  states  and  modes  of  the  asset.  One  emerging  trend  in  scalable  interface  design  is  the  use  of  Personal 
Digital  Assistants  (PDA).  These  hand-held  devices  provide  a  touch  screen  that  can  display  video,  a  thumb  pad,  and  four 
of  more  physical  buttons.  They  can  provide  limited  asset  control  as  compared  to  a  full  scalable  interface,  but  they  can 
provide  enough  functionality  to  control  simple  mobility. 

5.7.  Haptic  &  Tactile 

Haptic  and  tactile  devices  make  use  of  the  sense  of  touch.  Tactile  is  a  specific  type  of  haptic  device  that  uses  pressure 
or  vibration  stimulators  that  interact  with  the  skin.  An  example  of  a  haptic  device  is  the  “vibrate”  function  found  on 
most  pagers  and  cell  phones.  Haptic  devices  of  the  greatest  potential  interest  to  the  U.S.  Army  include  tactile  displays 
that  provide  information  through  the  use  through  some  type  of  sensor  emplaced  on  the  skin  or  in  arrays  on  a  vest. 
Because  skin-emplaced  sensors  can  signal  events,  tactile  displays  can  be  used  to  provide  information  when  visual  or 
audio  cues  may  not  be  available.  Tactile  display  information  includes  warnings  and  alarms,  navigation  and  guidance 
cues,  system  location,  and  malfunction  information,  and  threat  warnings. 

5.8.  Biometric 

Biometric  technologies  are  chiefly  for  security  purposes,  allowing  the  proper  level  of  interaction  via  authentification 
with  the  system  and  accords  the  user  required  access  levels.  Examples  of  this  are  fmgerprint/thumbprint  signatures, 
retinal  scans,  face  and  voice  recognition.  This  technology  is  useful  in  presenting  the  user  with  proper  data  upon 
accessing  the  system,  and  will  be  explored  for  utility  in  human-robot  applications. 

5.9.  Gesture 

Gesturing  is  a  promising  technology  for  intuitively  interfacing  with  computer  systems,  and  is  especially  useful  where 
silent  operations  are  desired  or  where  space  is  limited  (ex.  keyboard  stowage  and  power  requirements).  This  technology 


will  be  evaluated  as  a  potential  candidate  in  optimizing  human  interfacing  with  given  mounted  and  dismounted 
controller  devices. 

5.10.  Face  Recognition 

Face  recognition  is  a  technology  that  can  be  used  for  security  purposes,  but  can  also  be  utilized  for  interpreting  user 
intent.  Under  this  category,  lip  reading  and  eye  tracking  are  also  included.  These  technologies  will  be  explored  for 
potential  application  in  the  HRI  ATO,  but  have  not  been  analyzed  in  depth. 

5.11.  Human  Feedback/Stress  Monitors 

In  order  to  properly  analyze  human  workload  levels,  non-intrusive  technologies  are  necessary  to  assess  whether  the 
soldier  is  overloaded  and  must  shed  tasks  or  have  specific  tasks  automated.  Current  human  feedback  and  stress 
monitoring  devices  are  being  assessed  for  this  requirement,  and  will  be  explored  in  detail  to  determine  effectiveness  in 
assessing  soldier  health  and  workload. 


6.  CONCLUSION 

The  HRI  ATO  has  established  a  systems  engineering  approach  and  modeling  methodology  for  addressing  the  barriers 
identified  by  the  Army  Science  Board.  This  approach,  when  combined  with  the  proper  mix  of  technologies  scaled  for 
mounted  and  dismounted  control  and  utilization  of  recursive  testing  to  validate  the  models  developed,  will  yield 
optimum  results  for  mounted  and  dismounted,  ground  and  air  soldier-robot  teams.  The  end  goal  is  reduced  training  for 
the  soldier  across  varying  missions,  reduced  overall  workload  in  performing  these  missions,  and  specifically  reducing 
the  robot  controlling  portion  of  the  mission  to  allow  the  soldier  to  concentrate  on  his  primary  mission. .  .defending  our 
nation. 
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